Learning to Walk with Model Assisted Evolution Strategies

Many algorithms in robotics contain parameterized models. The setting of the parameters in general has a strong impact on the quality of the model. Finding a parameter set which optimizes the quality of the model typically is a challenging task, especially if the structure of the problem is unknown and can not be specified mathematically, i.e. the only way to get the function value for a certain parameter set is to test them on the real problem. Controlling the robot’s legs in order to walk is an example of such kind of black box approach. The walk is usually defined by three-dimensional trajectories for the movement of the legs and feet. The parameters (e.g. step height, step length, timings etc.) of these trajectories affect the stability and speed of the walking gait. Finding a parameter set which creates an optimal (in this case fast) walk can not be done by hand, since the direct effect of a parameter change is unclear. This chapter describes how Evolution Strategies can be used for the gait optimization of a four-legged robot whose walk is defined by 31 parameters and presents how Model Assisted Evolution Strategies lead to a faster convergence of the optimization of the problem. At first an introduction to the standard Evolution Strategy with self-adaptation of the mutation strengths is given. The individuals of the Evolution Strategy here define a set of walk parameters. To evaluate the fitness of each individual, the robot has to walk for a certain time with this parameter setting and measure the resulting walk speed. This is not only time consuming but also causes a lot of wear out of the robot – especially parameters resulting in stumbling or tumbling can damage the robot. Consequently, the goal is to minimize the amount of real fitness evaluations on the robot. For that reason Model Assisted Evolution Strategies will be introduced. These strategies use the fitness measurements as obtained samples from the search space to predict the fitness of the individuals and – based on the model – sort out the least promising individuals, resulting in a faster convergence of the optimization process. Since the quality of the model used for fitness prediction has a big effect on the convergence rate of the Model Assisted Evolution Strategy, mechanisms for controlling the impact of the model will be explained. The chapter closes with the results and comparisons of the walk learning process of the standard Evolution Strategy and its model assisted modifications.